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United States Government Acconntability Office 
Washington, DC 20548 



July 24, 2009 

The Honorable Robert C. Scott 
Chairman 

Subcommittee on Crime, Terrorism, 
and Homeland Security 
Committee on the Judiciary 
House of Representatives 

Subject: Juvenile Justice: Technical Assistance and Better Defined Evaluation Plans 
Will Help to Improve Girls’ Delinquency Programs 

Dear Mr. Chairman: 

Girls’ delinquency has attracted the attention of federal, state, and local policymakers for 
more than a decade as girls have increasingly become involved in the juvenile justice 
system. For example, from 1995 through 2005, delinquency caseloads for girls in juvenile 
justice courts nationwide increased 15 percent while boys’ caseloads decreased by 12 
percent. Also, from 1995 through 2005, the number of girls’ cases nationwide involving 
detention increased 49 percent compared to a 7 percent increase for boys.^ More 
recently, in 2007, 29 percent of juvenile arrests — about 641,000 arrests — involved girls, 
who accounted for 17 percent of juvenile violent crime arrests and 35 percent of juvenile 
property crime arrests.^ Further, in a 2007 survey of states conducted by the Federal 
Advisory Committee on Juvenile Justice, 22 states listed girls’ delinquency as an issue 
affecting their states’ juvenile justice systems.® State justice officials responding to the 
survey noted that juvenile female offenses have increased sharply and also noted that 
juvenile female offenders generally had more serious and wide-ranging service needs 
than juvenile male offenders, including treatment for substance abuse and mental health 
conditions. 



'C. Puzzanchera and W. Kang, Juvenile Court Statistics Databook (2008), 
http://oiidp.ncirs.gov/oistatbb/icsdb/ (accessed June 30, 2009). Most current data available. 

^C. Puzzanchera, Juvenile Arrests 2007, (2009) www.ncirs.gov/pdffilesl/oiidp/225344.pdf (accessed June 
26, 2009). 

^The Federal Advisory Committee on Juvenile Justice is an advisory body established by the Juvenile 
Justice and Delinquency Prevention Act of 1974, as amended, to advise the President and Congress on 
state perspectives regarding the operation of the Office of Juvenile Justice and Delinquency Prevention 
and on federal legislation pertaining to juvenile justice and delinquency prevention, to advise the 
Administrator of the Office of Juvenile Justice and Delinquency Prevention, and to review federal policies 
regarding juvenile justice and delinquency prevention. 42 U.S.C. § 5633(f). The Federal Advisory 
Committee on Juvenile Justice comprises appointed representatives from each of the 50 states, the District 
of Columbia, and the 5 U.S. territories. 
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The Office of Juvenile Justice and Delinquency Prevention (OJJDP) is the Department of 
Justice (DOJ) office charged with providing national leadership, coordination, and 
resources to prevent and respond to juvenile delinquency and victimization. OJJDP 
supports states and communities in their efforts to develop and implement effective 
programs to, among other things, prevent delinquency and intervene after a juvenile has 
offended. For example, from fiscal years 2007 through 2009, Congress provided OJJDP 
almost $1.1 billion for grants to states, localities, and organizations for a variety of 
juvenile justice programs. In support of this mission, the office also funds research and 
program evaluations related to a variety of juvenile justice issues, including girls’ 
delinquency. 

As programs have been developed at the state and local levels in recent years that 
specifically target preventing girls’ delinquency or intervening after girls have become 
involved in the juvenile justice system, it is important that agencies providing grants and 
practitioners operating the programs have information about which of these programs 
are effective. In this way, agencies can help to ensure that limited federal, state, and local 
funds are well spent. In general, effectiveness is determined through program 
evaluations, which are systematic studies conducted to assess how well a program is 
working — that is, whether a program produced its intended effects. To help ensure that 
grant funds are being used effectively, you asked us to review OJJDP’s efforts related to 
studying and promoting effective girls’ delinquency programs. This report addresses the 
following questions: 

1. What efforts, if any, has OJJDP made to assess the effectiveness of girls’ 
delinquency programs? 

2. To what extent are OJJDP’s efforts to assess girls’ delinquency programs 
consistent with generally accepted social science standards and the internal 
control standard to communicate with external stakeholders? 

3. What are the findings from OJJDP’s efforts to assess the effectiveness of girls’ 
delinquency programs, and how, if at all, does OJJDP plan to address the findings 
from these efforts? 

To identify OJJDP’s efforts to assess the effectiveness of girls’ delinquency programs, we 
analyzed relevant laws related to the office’s role in supporting research and evaluations 
on delinquency programs. We also analyzed OJJDP budget data for fiscal years 2007 
through 2009. We chose these years because they provide the most recent overview of 
the funding the office has had available to support its evaluation activities. We examined 
reports from research organizations and academic journal articles on girls’ delinquency 
issues. In our review, we focused on OJJDP’s efforts related to programs that are 
specifically designed for girls, not programs designed for both girls and boys. To identify 
OJJDP’s efforts, we reviewed a list of its grants to fund studies of girls’ delinquency 
programs from 1998 to 2008. We chose this time frame, the past 10 years from the start of 
our work, because it provided us with an overview of OJJDP’s efforts related to 
assessing girls’ delinquency programs. We also analyzed documentation about OJJDP’s 
establishment of a study group on girls’ delinquency issues, including the program 
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announcement and cooperative agreement/ We interviewed OJJDP officials, including 
the research coordinator who managed the study group project, about the office’s role in 
overseeing the group’s research. We also interviewed the current and former principal 
investigators of the study group project regarding the formation of the group, its 
activities, and its methodologies. To gather information on OJJDP’s efforts, we 
conducted interviews with 18 girls’ delinquency subject matter experts, that is, 
researchers and practitioners. We selected these experts based on their knowledge and 
experience with girls’ delinquency issues, which we determined through our review of 
the literature and from suggestions of experts to interview from study group members 
and OJJDP. These 18 experts included 11 of the 15 study group members and 7 experts 
who were not members of the group.*’ While their comments cannot be generalized to all 
girls’ delinquency experts, we nonetheless believe that their views gave us useful insights 
on issues related to girls’ delinquency and OJJDP’s efforts to assess girls’ programs. 

To determine the extent to which OJJDP’s efforts to assess girls’ delinquency programs 
were consistent with generally accepted social science standards, we reviewed the 
criteria the study group used to assess studies of girls’ delinquency programs and 
whether the group’s application of those criteria was consistent with standards for 
evaluation research.’^ To determine the extent to which these OJJDP efforts were 
consistent with the internal control standard to communicate with external stakeholders, 
we compared the office’s efforts with criteria in Standards for Internal Control in the 
Federal Government, specifically that agency management should ensure that there are 
adequate means of obtaining information from and communicating with external 
stakeholders who may have a significant impact on the agency achieving its goals.® We 
reviewed documentation about the composition of the study group and the criteria used 
to select the group members, such as their relevant fields of expertise, knowledge, and 
experience with girls’ issues. We also examined the study group’s external 
communications efforts, including its Web site, findings bulletins, conference 



^Cooperative agreements, rather than grant awards, can be used by federal agencies when substantial 
involvement is expected between the agency and the recipient when carrying out the activities described 
in the program announcement. 

^GAO defines an expert as a person who is recognized by others who work in the same subject matter area 
as having knowledge that is greater in scope or depth than that of most people working in that area. The 
expert’s knowledge can come from education, experience, or both. We specifically identified researchers 
who focus on girls’ delinquency issues and practitioners who operate programs that address girls’ 
delinquency. 

"^We contacted all 15 of the study group members. However, 1 member declined to be interviewed, and 3 
study group members did not respond to requests for interviews. 

Vor social science standards for evaluation research, see Donald T. Campbell and Julian Stanley, 
Experimental and Quasi-Experimental Designs for Research (Chicago: Rand McNally, 1963); William R. 
Shadish, Thomas D. Cook, and Donald T. Campbell, Experimental and Quasi-Experimental Designs for 
Generalized Causal Inference (Boston: Houghton Mifflin, 2002); Carol H. Weiss, Evaluation: Methods for 
Studying Programs and Policies, Second Edition (Englewood Cliffs, N.J.: Prentice-Hall, Inc., 1998); and 
GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.: May 1991). 

“GAO, Standards for Internal Control in the Eederal Government, GAO/AIMD-00-21.3.1 (Washington, 

D.C.: November 1999). 
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presentations, academic journal articles, and published book. In addition, we 
interviewed OJJDP officials about these dissemination efforts, as well as 18 girls’ 
delinquency experts regarding their views on the composition of the study group. 

To determine the findings from OJJDP’s efforts to assess the effectiveness of girls’ 
delinquency programs, and to assess how, if at ah, OJJDP plans to address these 
findings, we analyzed documentation such as published bulletins and conference 
presentations about the study group’s findings and recommendations related to program 
effectiveness. We also interviewed OJJDP officials knowledgeable about the office’s 
planning efforts and the current and former study group principal investigators regarding 
the group’s findings and recommendations. We compared OJJDP’s stated plans with 
criteria in standard practices for program management.® 

We conducted this performance audit from July 2008 through July 2009 in accordance 
with generally accepted government auditing standards. Those standards require that we 
plan and perform the audit to obtain sufficient, appropriate evidence to provide a 
reasonable basis for our findings and conclusions based on our audit objectives. We 
believe that the evidence obtained provides a reasonable basis for our findings and 
conclusions based on our audit objectives. 

Results in Brief 

To assess the effectiveness of girls’ delinquency programs, OJJDP established the Girls 
Study Group (Study Group). With an overall goal of developing research that 
communities need to make sound decisions about how best to prevent and reduce girls’ 
delinquency, the Study Group was established in 2004 under a $2.6 million multiyear 
cooperative agreement with a research institute. OJJDP’s objectives for the group, 
among others, included identifying effective or promising programs, program elements, 
and implementation principles (i.e., guidelines for developing programs) and developing 
program models to help inform communities of what works in preventing or reducing 
girls’ delinquency; identifying gaps in girls’ delinquency research and developing 
recommendations for future research; and disseminating findings to the girls’ 
delinquency field about effective or promising programs. To meet OJJDP’s objectives, 
among other things, the Study Group identified studies of delinquency programs that 
specifically targeted girls. The group then assessed the methodological quality of the 
studies using a set of criteria developed by DOJ’s Office of Justice Programs (OJP) called 
What Works to determine whether the studies provided credible evidence that the 
programs were effective at preventing or responding to girls’ delinquency. “ 



‘"Program management standards we reviewed are reflected in the Project Management Institute’s The 
Standard for Program Management © (2006). 

“The What Works criteria define six levels of effectiveness, including effective, promising, and ineffective, 
for use in assessing and classifying studies on the basis of their evidence of effectiveness. Additional 
details on these criteria are discussed later in this report. 
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OJJDP’s effort to assess girls’ delinquency programs through the use of a study group 
and the group’s methods for assessing studies were consistent with generally accepted 
social science research practices and standards, and OJJDP’s efforts to involve 
practitioners in Study Group activities and disseminate findings were also consistent 
with the internal control standard to communicate with external stakeholders/^ 



• According to OJJDP officials — including the research coordinator — they formed 
the Study Group rather than funding individual studies of programs because study 
groups provide a cost-effective method of gaining an overview of the available 
research in an issue area. As part of its work, the group collected, reviewed, and 
analyzed the methodological quality of research on girls’ delinquency programs. 
The use of such a group, including its review, is an acceptable approach for 
systematically identifying and reviewing research conducted in a field of study. 
This review helped consolidate the research and provide information to OJJDP for 
determining evaluation priorities. Further, we reviewed the criteria the group 
used to assess the studies and found that they adhere to generally accepted social 
science standards for evaluation research. We also generally concurred with the 
group’s assessments of the programs based on these criteria. According to the 
group’s former principal investigator, the Study Group decided to use the What 
Works criteria to ensure that its assessment of program effectiveness would be 
based on highly rigorous evaluation standards, thus eliminating the potential that 
a program that may do harm would be endorsed by the group. However, 8 of the 
18 experts we interviewed said that the criteria created an unrealistically high 
standard, which caused the group to overlook potentially promising programs. 
OJJDP officials stated that despite such concerns, they approved the group’s use 
of the criteria because of the methodological rigor of the framework and their 
goal for the group to identify effective programs. 

• In accordance with the internal control standard to communicate with external 
stakeholders, OJJDP sought to ensure a range of stakeholder perspectives related 
to girls’ delinquency by requiring that Study Group members possess knowledge 
and experience with girls’ delinquency and demonstrate expertise in relevant 
social science disciplines. The initial Study Group, which was convened by the 
research institute and approved by OJJDP, included 12 academic researchers and 
1 practitioner, a member with experience implementing girls’ delinquency 
programs. Eleven of the 18 experts we interviewed stated that this composition 
was imbalanced in favor of academic researchers, six of whom said that the 
composition led the group to focus its efforts on researching theories of girls’ 
delinquency rather than gathering and disseminating actionable information for 
practitioners.^^ According to OJJDP officials, they acted to address this issue by 
adding a second practitioner as a member and involving two other practitioners in 
study group activities. OJJDP officials stated that they plan to more fully involve 
practitioners from the beginning when they organize study groups in the future 
and to include practitioners in the remaining activities of the Study Group, such as 



“GAO/AIMD-00-21.3.1. 

‘^The other seven experts did not express views regarding the balance of the study group’s composition. 
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presenting findings at a national conference. Also, in accordance with the internal 
control standard, OJJDP and the Study Group have disseminated findings to the 
research community, practitioners in the girls’ delinquency field, and the public 
through conference presentations, Web site postings, and published bulletins and 
plan to report on all of the group’s activities by spring 2010. 

To address the Study Group findings that few girls’ delinquency programs had been 
studied and that the available studies lacked conclusive evidence of program 
effectiveness, OJJDP plans to provide technical assistance to help programs be better 
prepared for evaluation; however, more fully developing plans for supporting evaluations 
could help OJJDP address its girls’ delinquency goals. The Study Group found that the 
majority of the girls’ delinquency programs it identified — 44 of the 61 — had not been 
studied by researchers. For the 17 programs that had been studied, the Study Group 
reported that none of the studies provided conclusive evidence with which to determine 
whether the programs were effective at preventing or reducing girls’ delinquency. For 
example, according to the Study Group, 11 of the 17 studies lacked evidence of program 
effectiveness because, for instance, the studies involved research designs that could not 
demonstrate whether any positive outcomes, such as reduced delinquency, were due to 
program participation rather than other factors. Based on the results of this review, the 
Study Group reported that among other things, there is a need for additional, 
methodologically rigorous evaluations of girls’ delinquency programs; training and 
technical assistance to help programs prepare for evaluations; and funding to support 
girls’ delinquency programs found to be promising. According to OJJDP officials, in 
response to the Study Group’s finding about the need to better prepare programs for 
evaluation, the office plans to work with the group and use the remaining funding from 
the effort — approximately $300,000 — to provide technical assistance workshop in 
October 2009. The workshop is intended to help approximately 10 girls’ delinquency 
programs prepare for evaluation by providing information about how evaluations are 
designed and conducted, how to identify appropriate performance measures, and how to 
collect data that will be useful for program evaluators in assessing outcomes. In addition, 
OJJDP officials stated that as a result of the Study Group’s findings along with feedback 
they received from members of the girls’ delinquency field, OJJDP plans to issue a 
solicitation in early fiscal year 2010 for researchers to apply for funding to conduct 
evaluations of two to five girls’ delinquency programs. OJJDP has also reported that the 
Study Group’s findings are to provide a foundation for moving ahead on a comprehensive 
program related to girls’ delinquency. However, OJJDP has not developed a plan that is 
documented, is shared with key stakeholders, and includes specific funding 
requirements and commitments and time frames for meeting its girls’ delinquency goals. 
Standard practices for program and project management state that specific desired 
outcomes or results should be conceptualized, defined, and documented in the planning 
process as part of a road map, along with the appropriate projects needed to achieve 
those results, supporting resources, and milestones. In addition, government internal 
control standards call for policies and procedures that establish adequate 



‘^Project Management Institute, The Standard for Program Management. 
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communication with stakeholders as essential for achieving desired program goals/^ 
According to OJJDP officials, they have not developed such a plan because the office is 
in transition and is in the process of initiating efforts to develop an officewide research 
plan, but they are taking steps to address their girls’ delinquency goals, for example, 
through the workshop and planned evaluations. Developing such a plan would help 
OJJDP to demonstrate leadership to the girls’ delinquency field by clearly articulating the 
actions it intends to take to meet its goals and would also help the office to ensure that 
the goals are met. 

To help ensure that OJJDP meets its goals to identify effective or promising girls’ 
delinquency programs and supports the development of program models, we are 
recommending that the Administrator of OJJDP develop and document a plan that 

(1) articulates how the office intends to respond to the findings of the Study Group, 

(2) includes time frames and specific funding requirements and commitments, and (3) is 
shared with key stakeholders. In commenting on a draft of this report, OJP agreed with 
our recommendation and outlined efforts that OJJDP plans to undertake to respond to 
the findings of the Study Group, which we describe in the report. OJP comments are 
reprinted in the enclosure. 

Background 

Over the past two decades girls have increasingly become involved in the juvenile justice 
system, and while the majority of juvenile arrests and cases involve boys, research has 
indicated that girls have more intensive treatment needs than boys. In 1980, 20 percent of 
all juvenile arrests were girls; by the mid-1990s about one quarter of these arrests were 
girls; and by 2007, girls accounted for 29 percent of all juvenile arrests. Additionally, 
while arrests for some violent crimes, such as assaults, have decreased for males, they 
have decreased less, or in some cases have increased, for females. For example, between 
1998 and 2007 juvenile male arrests for simple assault declined 4 percent, and female 
arrests increased 10 percent. Further, from 1985 through 2005, the estimated number of 
girls’ delinquency cases involving detention increased by 92 percent, and those cases that 
involved probation increased by 88 percent. Research on girls has highlighted that 
delinquent girls have higher rates of mental health problems than delinquent boys, 
receive fewer special services, and are more likely to abandon treatment programs. For 
example, one study showed that detained girls have more symptoms of mental illness 
than would be predicted on the basis of gender or setting alone. Research has also 
shown that delinquent girls have higher mortality rates, dysfunctional and violent 
relationships, poor educational achievement, and less stable work histories than 



‘'‘GAO/AIMD-00-21.3.1. 

‘®The Study Group found that possible reasons for increased arrest rates for girls include changes in local 
law enforcement policies that lowered the threshold for reporting assaults or categorizing assaults as 
aggravated, reclassification of domestic dispute offenses as simple assaults that can result in arrest, and 
increased referrals to police resulting from schools’ zero tolerance policies for violence. 

“Elizabeth Cauffman and others, “Gender Differences in Mental Health Symptoms among Delinquent and 
Community Youth,” Youth Violence and Juvenile Justice, vol. 5, no. 3 (2007): 287-307. 
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nondelinquent girls. Further, girls’ delinquency has been linked to drug abuse, mental 
health problems and disorders, poorer physical health, and victimization by and violence 
toward partners in adulthood. 

In recent years, programs have been developed that specifically target preventing girls’ 
delinquency and intervening once girls have become involved in the juvenile justice 
system. In general, prevention programs provide services and programming, such as 
substance abuse education, mentoring, and life skills education, to deter girls from 
becoming involved in criminal or other antisocial activities. Intervention programs 
provide services to girls once they have entered the juvenile justice system, for example, 
through programs that are alternatives to probation or that provide intensive services for 
girls who are on probation, to prevent them from returning to the system or entering the 
adult criminal justice system. These services could include visits by probation officers, 
individual case plans, substance abuse treatment and therapy, funds for emergency 
situations, life skills courses, teen pregnancy services, and therapy sessions. 

The Juvenile Justice and Delinquency Prevention Act (the Act) established OJJDP in 
1974.^^^ As the only federal office charged exclusively with preventing and responding to 
juvenile delinquency and victimization and with helping states improve their juvenile 
justice systems, OJJDP supports its mission through a variety of activities, including: 
funding research and evaluation efforts, statistical studies, and demonstration programs; 
providing training and technical assistance; producing and distributing publications and 
other products containing information about juvenile justice topics; and administering a 
wide variety of grants to states, territories, localities, and public and private 
organizations through formula, block, and discretionary grant programs.^® Table 1 shows 
OJJDP’s enacted appropriations for fiscal years 2007 through 2009. 



"42 U.S.C. §5611. 

'®OJJDP allocates some formula and block grants to states on the basis of states’ juvenile populations, 
while others may be awarded on the basis of a fixed level to all states. Discretionary grants are generally 
awarded through a competitive process to state and local governments as well as individual agencies and 
organizations. Under the Act, “state” means any of the United States, the District of Columbia, the 
Commonwealth of Puerto Rico, the Virgin Islands, Guam, American Samoa, and the Commonwealth of the 
Northern Mariana Islands. 42 U.S.C. § 5603. 
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Table 1 : Juvenile Justice Appropriations Fiscai Years 2007 through 2009 



Dollars In thousands 


Line Item 


Funding by fiscal year 






2007 


2008 


2009 


Part A - Concentration of Federal Efforts” 


$703 


$658 


$0 


Part B - State Formula Grants 


78,976 


74,260 


75,000 


Part D — Research, Evaluation, Technical 
Assistance and Training 


0 


0 


0 


Part E - Developing, Testing and 
Demonstrating Promising New Initiatives and 
Projects 


104,670 


93,835 


82,000 


Youth Mentoring Grants 


9,872 


70,000 


80,000 


Title V - Local Delinquency Prevention 
Incentive Grants 


64,168 


61,100 


62,000 


Project Childsafe" 


987 


0 


0 


Secure Our Schools 


14,808 


15,040 


0 


VOCA — Improving Investigation and 
Prosecution of Child Abuse Program 


14,808 


16,920 


20,000 


Juvenile Accountability Block Grant Program" 


49,360 


51 ,700 


55,000 


Total 


$338,352 


$383,513 


$374,000 



Sources: Revised Continuing Appropriations Resoiution, 2007, Pub. L. No. 1 1 0-5, 1 21 Stat. 8, 8-9 (inciuding the across-the-board rescission of 1 .28 percent provided in the 
continuing resolution): Consoiidated Appropriations Act, 2008, Pub. L No. 110-161, 121 Stat. 1844, 1911-12 (2007); and Omnibus Appropriations Act, 2009, Pub. L. No. 111-8, 
123 Stat. 524, 581-82. 

‘^According to OJP's fiscal year 2010 congressional budget submission, the Concentration of Federal Efforts program 
promotes interagency cooperation and coordination among federal agencies with responsibilities in the area of 
juvenile justice, as authorized by Part A of the Act, as amended. 

‘’Project Childsafe is a nationwide program to promote safe firearms handling and storage practices through the 
distribution of safety education messages and free gun-locking devices. 

“Under the Juvenile Accountability Block Grant Program, OJJDP provides funds to states and units of local 
government for the purpose of strengthening the juvenile justice system. These funds can be used for 17 different 
purpose areas, including establishing programs to help the successful reentry of juvenile offenders from state and 
local custody in the community or for hiring or training programs for detention and corrections personnel. 

OJJDP, through its various grant programs, has provided funding to states and 
organizations to support girls’ delinquency programs, although it is not specifically 
required by the Act to fund such programs in particular. For example, to be eligible to 
receive formula grants, states are required to submit a plan to OJJDP for providing 
gender-specific services for juvenile delinquency prevention and treatment.^** However, 
the states generally have the authority to determine how formula and block grants are 
allocated and may use these funds to support a range of program areas, including 
programs specifically for delinquent girls. For example, for fiscal years 2007 and 2008, 
OJJDP reported that states used approximately $1.9 million in Part B formula grant 
money for girls’ delinquency programs, representing approximately 1 percent of such 
funding for those years. In addition, in fiscal year 2007, OJJDP reported awarding about 
$1.8 million in discretionary grant awards to prevention and intervention programs 
addressing girls’ delinquency.^” 



“42 U.S.C. § 5633(a)(7)(B). 

“OJJDP did not report awarding discretionary grants for girls’ programs in fiscal year 2008, and as of June 
2009, OJJDP had not awarded fiscal year 2009 discretionary grants. 
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The Act requires the OJJDP Administrator to conduct and support evaluations and 
studies of the performance and results achieved by federal juvenile delinquency 
programs and activities, although the law does not specifically require OJJDP to fund 
evaluations of state or locally funded programs or those specifically focused on girls’ 
delinquency.^^ OJJDP has provided funding for evaluations using (1) funds appropriated 
for Part D of the Act — which allows the Administrator to conduct research and 
evaluation, information dissemination, and training and technical assistance,^^ or 
(2) funds set aside from several of its appropriations accounts for use for research, 
evaluation, and statistics activities.^® Funding has not been appropriated to OJJDP for 
Part D since fiscal year 2005 when it received $10 million, so OJJDP has allocated 
funding for research and evaluation of programs from fiscal years 2006 through 2008 
using approximately $40 million in funding from appropriation set asides.®^ 

OJJDP has provided funding for several efforts designed to provide information about 
girls’ delinquency programs to the juvenile justice field in the past decade. For example, 
in 1998, the office published an inventory of best practices that included a list of 16 
promising girls’ delinquency programs, which had been compiled by a research 
organization as part of a $1.1 million cooperative agreement to provide training and 
technical assistance to states and localities about girls’ programs.®® The research 
organization identified these 16 programs on the basis of programmatic criteria — such as 
whether the program used appropriate assessments to determine treatment plans; 
provided empowerment strategies, such as skill training and vocational training; or 
provided its staff with gender-specific training — rather than on whether the program’s 
effectiveness had been studied by researchers. Further, this effort found that more 
research was needed to draw conclusions about the effectiveness of girls’ delinquency 
program models. In addition, during this time OJJDP spent approximately $1.1 million to 
fund four studies of girls’ delinquency issues. While these studies assessed issues related 
to girls’ delinquency, they did not specifically assess the effectiveness of girls’ 
delinquency programs. For example, in 2000, OJJDP funded one study of women in 
gangs, which found, among other things, that the optimum time for prevention and 



^‘42 U.S.C. § 5614(b)(3). 

=^"42 U.S.C. §§ 5661-62. 

Appropriations statutes for fiscal years 2006 through 2008 provided that OJJDP may use not more than 10 
percent of each amount appropriated for research, evaluation, and statistics activities that benefit the 
programs or activities authorized, and not more than 2 percent of each appropriated amount for training 
and technical assistance. See, e.g, Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 
1844, 1906-07 (2007). This provision applied to appropriation accounts under Juvenile Justice Programs, 
but did not apply to amounts appropriated for demonstration projects, as authorized by sections 261 and 
262 of the Act, 42 U.S.C. §§ 5665-66. 

^'‘The $40 million comprises set asides eligible to be used for research and evaluation of girls’ delinquency 
programs. As of July 2009, OJJDP has not determined how it would use its fiscal year 2009 appropriation 
set asides. 

^^Greene, Peters Associates, Guiding Principles for Promising Female Programming: An Inventory of 
Best Practices (Washington, D.C.: Office of Juvenile Justice and Delinquency Prevention, 1998). 
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intervention was the middle teen years and that the optimum place for intervention was 
school before girls drop out. Another study compared three treatment models to 
determine which was most effective at reducing the number of institutional placements 
for adjudicated female offenders. The study found that girls with the most serious and 
frequent crises were more dissatisfied with social services or were denied access to such 
services. The study highlighted the importance of youth assistance programs to provide 
opportunities for girls to develop pro-social skills through family, school, and community 
connections. 

OJJDP Established the Girls Study Group to Assess the Effectiveness of Girls’ 
Delinquency Programs 

OJJDP initiated the Study Group to assess the effectiveness of girls’ delinquency 
programs. In response to increases in girls’ arrests through the 1990s and early 2000s and 
questions about the causes of these increases and how best to respond to the needs of 
girls entering the juvenile justice system, OJJDP issued a program announcement in 2003 
for a study group to focus on girls’ delinquency issues.^ While OJJDP had funded studies 
on girls’ issues and a technical assistance effort to assist girls’ delinquency programs in 
their operations, in forming the Study Group, OJJDP determined that a comprehensive, 
research-based foundation was needed to guide state and local policymakers and 
practitioners in their efforts to effectively prevent and reduce girls’ delinquency. In its 
announcement for the Study Group, OJJDP highlighted the need for more information 
about female development and female-specific delinquency risk factors, as well as the 
effectiveness of girls’ delinquency programs to ensure the best services and treatment. 
OJJDP sought applications from public and private organizations to convene a study 
group to address these issues and in 2004 awarded a 2-year cooperative agreement to 
Research Triangle Institute (RTI) to do so. OJJDP has since provided RTI with an 
extension through June 2010 to complete all of the Study Group’s activities. The total 
funding awarded for the cooperative agreement was almost $2.6 million. 

OJJDP articulated five broad objectives for the Study Group in its September 2003 
program announcement. Three of these objectives specifically related to assessing and 
promoting girls’ delinquency programs: (1) identifying effective or promising programs. 



^ Jn 2000, OJJDP issued program announcements for two separate girls’ delinquency efforts — a study group 
and a girls’ institute. The first effort, a girls study group, was awarded to a university in 2001. However, 
because it was unable to reach agreement on project management issues, the university terminated the 
agreement in 2002. In 2003, OJJDP reissued the program announcement for a girls study group and revised 
the announcement to clearly delineate the level of expected federal involvement, for example, by explicitly 
stating that OJJDP planned to review and approve all project consultants, plans, and products developed. 
The second effort, a national girls’ institute, was intended to put the study group’s findings into practice by, 
among other things, promoting programs for girls; providing training and technical assistance to the field 
on girls’ delinquency issues; facilitating coordination among federal, state, and local organizations serving 
girls; and disseminating information about the research findings of the study group. According to OJJDP 
officials, the 2000 announcement never received funding, and OJJDP did not reissue it in later years 
because of funding constraints. Instead, when OJJDP reissued the program announcement in 2003 for a 
study group, it incorporated elements of the planned institute. For example, the 2003 study group 
solicitation included objectives for identifying and promoting programs for girls and for disseminating 
information to the practitioner field. 
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program elements, and implementation principles to help inform communities about 
what works in preventing or reducing girls’ delinquency and to support the development 
of these program models; (2) identifying gaps in girls’ delinquency research and 
developing recommendations for future research to fill these gaps; and (3) disseminating 
findings to the girls’ delinquency field about effective or promising programs. The other 
two objectives included understanding the trends and consequences related to girls’ 
delinquency and developing a comprehensive theory of girls’ delinquency.^^ 

To meet OJJDP’s program assessment objectives, among other activities, the Study 
Group conducted a review of the literature on girls’ delinquency that included over 1,000 
documents in relevant research areas, such as criminological and feminist explanations 
for girls’ delinquency, patterns of delinquency, and the justice system’s response to girls’ 
delinquency. To identify girls’ delinquency programs, from June 2005 through October 
2006, the Study Group analyzed the results of this literature search, conducted Web 
searches, reviewed juvenile justice 3-year plans from 2000 to 2004 for all 50 states, 
reviewed federal agency and private organization lists of delinquency programs, and 
solicited suggestions on its Web site.^® The Study Group initially set out to identify 
federally funded girls’ delinquency programs but expanded its search to include state and 
locally funded programs after it found few federally funded programs. As a result, the 
Study Group identified 61 programs that specifically targeted preventing or responding 
to girls’ delinquency. The group then determined which of these programs had been 
studied for program effectiveness by conducting Web searches for evaluation materials 
and published research, reviewing abstracts from academic journals, contacting program 
directors, and reviewing program Web sites. 

To identify effective programs, the Study Group reviewed the studies of girls’ 
delinquency programs that it identified and classified them based on evidence of their 
effectiveness. To make this determination, the Study Group compared the studies’ 
methodologies to criteria established in the OJP What Works classification framework, 
which defines six levels of evidence of effectiveness, which are effective, effective with 
reservation, promising, and ineffective, as well as inconclusive evidence and insufficient 
evidence, as described in table 2.^** 



^’Specifically, the Study Group’s objective to understand the trends and consequences of girls’ delinquency 
involved increasing research-based knowledge about the risk and protective factors related to girls’ 
delinquency and determining the patterns and consequences of juvenile justice decisions on female 
offenders. The objective on developing a comprehensive theory of girls’ delinquency involved examining 
the extent to which theories developed primarily to explain boys’ delinquency applied to girls, as well as 
exploring whether theories that had been developed for girls were useful in developing and testing new 
prevention and intervention strategies. 

^*Under the Act, states are required to submit 3-year plans to OJJDP outlining their activities for investing 
in delinquency prevention and for coordinating services delivered to at-risk juveniles and their families, 
among other things. 42 U.S.C. § 5633. 

multiagency working group led by DOJ’s OJP, which included the Department of Health and Human 
Services and the Department of Education, developed the classification framework and criteria from 2004 
to 2005 to support a planned What Works repository to assist communities in selecting and replicating 
evidence-based programs that was never implemented. Federal government efforts to develop repositories 
of evidence-based programs have continued under Find Youth lr\fo, the Substance Abuse and Mental 
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Table 2: Summary of What Works Criteria Used by the Girls Study Group to Assess Studies of Girls’ 
Delinquency Programs 



Level of 
effectiveness 


Description 


Effective 


Effective programs have studies with a randomized controlled research design. These are designs 
that compare the outcomes for individuals that are randomly assigned either to the program or to 
a nonparticipating control group before the intervention in an effort to control for any systematic 
difference between the groups that could account for a difference in their outcomes. Effective 
programs also demonstrate a significant and sustained effect — that is, statistically significant 
positive outcomes that remain for at least 1 year after subjects stop participating in a program. 

The program should have been replicated at least once externally at another site to confirm 
results. 


Effective with 


These programs have studies with a randomized controlled research design that demonstrates a 


reservation 


significant and sustained effect. A program should have at least one replication to confirm results. 
Reservations occur either because the program has only an internal replication at the same site or 
because it has an external replication with modest results. 


Promising 


Promising programs have either studies with (1) a randomized controlled research design without 
a replication or (2) a quasi-experimental research design. These programs have significant and 
sustained effects. 


Insufficient 

evidence 


These are studies of programs that have a quasi-experimental research design that lack sufficient 
methodological rigor, or have a pre-post test design that involves tests that analyze measures 
before and after individuals participated in the program. 


Inconclusive 

evidence 


These studies of programs may have adequately rigorous research designs but not sustained 
effects, or they may have contradictory findings and not enough evidence demonstrating that the 
programs are either effective or ineffective. 


Ineffective 


These are studies of programs that have an experimental or quasi-experimental research design 
that failed to demonstrate a significant effect in an initial study or in a replication. 



Source: GAO analysis of OJP What Works criteria. 



According to the Study Group’s principal investigator, as of May 2009, the group had 
finalized its program review findings and was in the process of finishing a bulletin on 
these findings before providing it to OJJDP for publication. As of June 2009, OJJDP has 
issued three bulletins on several of the group’s activities. These bulletins have provided 
an overview of the Study Group’s activities and the group’s findings on its two objectives 
related to girls’ delinquency risk factors and patterns of offending. According to OJJDP 
officials, the Study Group plans to issue a final report that summarizes all of its activities 
and findings to OJJDP by spring 2010. 

OJJDP Efforts to Assess Program Effectiveness Were Consistent with Social 
Science Practices and Standards, and OJJDP Has Taken Action to Enhance Its 
Communication about Study Group Activities and Findings with External 
Stakeholders 

OJJDP’s efforts to assess program effectiveness through the use of a study group as well 
as the group’s efforts were consistent with generally accepted social science practices 
and standards, although experts we interviewed presented differing views on the criteria 
used to assess programs. OJJDP also took action to include external stakeholders in 



Health Services Administration’s National Registry of Evidence-Based Programs and Practices, and the 
OJJDP Model Programs Guide. Even though the repository was never implemented, the criteria within its 
framework are still valid to use in assessing evidence of program effectiveness. 
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study group activities and is disseminating the group’s findings consistent with standards 
for control in the federal government. 



The Use of a Study Group and the Group’s Efforts Were Consistent with Generally 
Accepted Social Science Practices and Standards: However. Experts We Interviewed 
Presented Differing Views on the Criteria Used to Assess Programs 

OJJDP’s efforts to assess girls’ delinquency programs, including its approach of using a 
study group and the group’s methods of assessing studies, were consistent with generally 
accepted social science standards for evaluation research. According to OJJDP officials, 
including the research coordinator, they chose to form a study group rather than fund 
individual evaluations of programs because study groups are a cost-effective method of 
gaining an overview of the available research in an issue area. As part of its work, the 
group collected, reviewed, and analyzed the methodological quality of research on girls’ 
delinquency programs. Such an approach of systematically identifying and reviewing 
research conducted in a field of study is an acceptable practice to consolidate the 
research in an area and provide information to enable program managers to determine 
where they might best commit future evaluation resources.^" Thirteen of the 18 girls’ 
delinquency experts we interviewed (including 1 1 Study Group members) stated that the 
Study Group’s efforts were useful for providing an overview of girls’ delinquency issues. 
However, 6 experts (including 2 Study Group members) also noted that it would have 
been beneficial to the girls’ delinquency field for the group to conduct evaluations to 
determine program outcomes or promising models rather than reviewing completed 
studies. OJJDP has funded individual studies of girls’ delinquency programs in the past 
but, according to OJJDP officials, was seeking to use the Study Group’s research to form 
a baseline of the available knowledge about girls’ delinquency issues. 

The Study Group’s effort to review the studies according to the What Works criteria was 
consistent with generally accepted social science standards. Specifically, we reviewed 
the OJP What Works criteria and found that they adhere to these standards for 
evaluation research. Using the What Works criteria, we also assessed the same studies 
for the 17 girls’ delinquency programs that the Study Group had reviewed and generally 
concurred with the Study Group’s ratings of the program studies. While the Study 
Group’s use of the What Works criteria was in keeping with social science standards, 
experts we interviewed expressed differing views on the group’s decision to use these 
criteria. According to the Study Group’s former principal investigator, the group decided 
to use the What Works criteria in 2005 because the criteria ensured that the group’s 
assessment of the effectiveness of programs in preventing or reducing girls’ delinquency 
would be based on highly rigorous evaluation standards to identify effective programs — 
thus eliminating the potential that a program that may do harm would be endorsed by the 
group. Eight Study Group members we interviewed also stated that the Study Group’s 
use of the criteria was appropriate because it ensured that the group would only 
disseminate information on programs determined to be effective based on a high level of 



“The approach used by OJJDP is similar to the evaluation synthesis methodology described in GAO, The 
Evaluation Synthesis, GAO/PEMD-10.1.2 (Washington, D.C.: March 1992). This type of approach might 
also be termed systematic review. 
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evidence. However, 8 other experts, including three Study Group members, said that the 
criteria created an unrealistically high standard, which caused the Study Group to 
overlook potentially promising programs.®^ Further, 9 of the 18 experts (including five 
Study Group members) we interviewed also noted that requiring a randomized 
controlled research design — a research design that compares the outcomes for 
individuals who are randomly assigned to either the program being studied or to a 
nonparticipating control group before the intervention — to demonstrate effectiveness, as 
the What Works criteria does, is a difficult standard to achieve because such a design is 
expensive, and programs may be reluctant to divert resources from programming to pay 
for evaluations. OJJDP officials stated that they understood the experts’ concerns and 
the trade-offs in using a classification framework that requires a randomized controlled 
research design to demonstrate effectiveness; however, they approved the group’s use of 
the criteria because it provided a rigorous framework for assessing program evaluations. 
We understand that studies can produce valid results using other research designs, such 
as studies using quasi-experimental designs or studies comparing the outcome results for 
groups of girls that are statistically matched. We have also previously reported that 
randomized controlled research designs provide researchers with the best method for 
assessing a program’s effectiveness — they isolate changes caused by the program from 
other factors — when doing so is feasible and ethical.^^ 

OJJDP Has Taken Actions to Reach Out to External Stakeholders on Study Group 
Activities and Findings and Is Disseminating the Findings in Keeping with Internal 
Control Standards 



OJJDP has taken action to reach out to external stakeholders to address concerns about 
the composition of the Study Group after its initial formation and, moving forward, plans 
to continue to incorporate program practitioners in its planned efforts. Standards for 
Internal Control in the Federal Government states that program managers should 
ensure that there are adequate means of obtaining information from and communicating 
with external stakeholders who may have a significant impact on the agency achieving 
its goals.®^ Regarding gaining information from external stakeholders, OJJDP’s program 
announcement for the Study Group sought to ensure a range of stakeholder perspectives 
related to girls’ delinquency. The announcement required that the members of the Study 
Group possess knowledge of and experience with female development and delinquent 
girls and demonstrate expertise in a variety of relevant social science disciplines, such as 
criminology, sociology, and developmental psychology. In awarding the cooperative 



^‘Two experts we interviewed did not express a view on the group’s approach to evaluating programs. 

^^GAO, Juvenile Justice: OJJDP Reporting Requirements for Discretionary and Formula Grantees and 
Concerns About Evaluation Studies, GAO-02-23 (Washington, D.C.: Oct. 30, 2001); Justice Outcome 
Evaluations: Design and Implementation of Studies Require More NIJ Attention, GAO-03-1091 
(Washington, D.C.: Sept. 24, 2003); Adult Drug Courts: Evidence Indicates Recidivism Reductions and 
Mixed Results for Other Outcomes, GAO-05-219 (Washington, D.C.: Feb. 28, 2005); and Abstinence 
Education: Assessing the Accuracy and Effectiveness of Eederally Eunded Programs, GAO-08-664T 
(Washington, D.C.: Apr. 23, 2008). 

'"GAO/AIMD-00-21.3.1. 
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agreement to RTI through a peer review process, OJJDP approved the RTI proposal for 
the Study Group as responding to the requirements and expectations of the program 
announcement. Consistent with the fields of expertise cited in the program 
announcement, RTI convened a group of 13 members, including 12 academic researchers 
from social science disciplines and one practitioner, a member directly involved in girls’ 
delinquency programming.^^ However, according to several of the experts we 
interviewed, this Study Group composition did not include sufficient representation and 
input from a key external stakeholders group — girls’ delinquency program practitioners. 
For example, 11 of the 18 girls’ delinquency experts we interviewed, including 5 study 
group members, said that the Study Group was imbalanced in favor of academic 
researchers, 6 of whom (including 2 study group members) said that the composition led 
the group to focus its efforts on researching theories of girls’ delinquency rather than 
gathering and disseminating actionable information for practitioners.^*^ According to 
OJJDP officials we interviewed, they had received feedback from girls’ delinquency 
stakeholders in 2006 on this issue. In response, according to OJJDP program managers, 
they acted to address the imbalance of the Study Group by adding a second practitioner 
as a member and involving 2 other practitioners in group activities, such as presenting 
successful girls’ delinquency program practices at conferences and reviewing the group’s 
work products.*^*^ OJJDP officials stated that as a lesson learned, they plan to more fully 
involve practitioners from the beginning when they organize study groups in the future. 

In addition, OJJDP officials noted that specific to the Study Group, they plan to continue 
to reach out to obtain information from and include practitioners in the remaining 
activities of the group, such as presenting findings at a national juvenile justice 
conference. 

OJJDP and the Study Group have disseminated the group’s findings to the research 
community, practitioners in the girls’ delinquency field, and the public in a variety of 
ways, and in doing so have made efforts to respond to stakeholder concerns. In its 2003 
program announcement, in keeping with the internal control standard for 
communicating with stakeholders, OJJDP required that the Study Group disseminate its 
findings through publications and products that address the needs of various practitioner 
audiences in diverse fields, including juvenile justice, child welfare, mental health, and 
substance abuse prevention. Since 2004, Study Group principal investigators and group 
members have presented findings at 24 conferences and posted the presentation slides to 
the group’s Web site.*^^ OJJDP has also published three bulletins on the Study Group’s 
activities and findings. Six girls’ delinquency experts we interviewed (including five 
Study Group members) stated that the information disseminated was generally helpful 
because it provided a useful overview of girls’ delinquency trends and research. 



^The Study Group members represented 11 of the 12 disciplines specified in the program armouncement. 

^“^The other seven experts did not express views regarding the balance of the study group’s composition. 

“in addition to the practitioner, OJJDP also added an expert in program evaluation as a group member 
after the group had begun its activities. 

“The Web site is located at http://girlsstudvgroup.rti.org/index.cfm?fuseaction=dsp home . 
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However, 10 of the 18 experts we interviewed (including three Study Group members) 
also noted that some of the group’s dissemination efforts created confusion among 
practitioners because Study Group members presented findings that did not 
acknowledge factors that practitioners believed contribute to girls’ delinquency, such as 
traumatic life experiences. According to OJJDP officials, in response to feedback they 
received from girls’ delinquency stakeholders about such concerns, the office and the 
Study Group sponsored workshop sessions at a conference for juvenile justice 
practitioners where the group clarified its findings and sought practitioner input on 
subjects such as delinquency risk and protective factors and trends in girls’ delinquency. 
According to OJJDP officials, the office and the Study Group plan to continue 
disseminating the group’s findings by issuing four additional bulletins and by presenting 
the findings at a national conference on juvenile delinquency. 

In Response to Study Group Findings of No Evidence of Effective Girls’ 
Delinquency Programs, OJJDP Plans Technical Assistance to Help Programs but 
Could Strengthen Its Plans for Supporting Evaluations 

The OJJDP-sponsored Study Group found that no programs in its review had evidence of 
effectiveness and, among other things, that additional support for program evaluation is 
needed. To address these findings, OJJDP plans to provide technical assistance to help 
girls’ delinquency programs so that they will be better prepared to be evaluated. 
However, by articulating time frames and specific funding requirements and 
commitments in its plans to support evaluations, OJJDP could better address its goals 
for preventing and reducing girls’ delinquency. 

The Study Group Found No Evidence of Effective Girls’ Delinquency Programs to 
Promote as Models and, among Other Things. That Evaluation Is Needed 

In its review of girls’ delinquency programs, the Study Group’s findings showed that the 
majority of the programs it identified — 44 of 61 — had not been studied by researchers, 
while 17 of the programs had been the subject of published studies. The Study Group 
determined that none of the 17 programs that had been studied had conclusive evidence 
of their effectiveness. Specifically, the Study Group found that the studies provided 
insufficient evidence of the effectiveness of 11 of these 17 programs. For example, our 
review of one study that the Study Group assessed as having insufficient evidence 
showed that the study had a quasi-experimental design but lacked methodological rigor 
in that the treatment and comparison groups had small sample sizes and did not appear 
to be well matched, and any statistical tests reported were only performed on treatment 
group participants.^® The Study Group found that for the remaining 6 programs, the 
studies provided inconclusive evidence of effectiveness. For example, our review of one 
study that the group assessed as having inconclusive evidence showed statistically 
significant results for the program; however, sustained effects were not indicated for at 
least a 1-year period beyond the end of the intervention. Further, it was unclear whether 



quasi-experimental design is a controlled study where study participants are assigned in a nonrandom 
manner to a treatment group (individuals participating in the program being studied) or a comparison 
group (individuals closely resembling those in the treatment group on many demographic variables but not 
participating in the program). 
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the study participants were representative of the population of girls that the program 
was designed to reach. As a result, there was not enough evidence to demonstrate that 
the program was either effective or ineffective for the intended population of delinquent 
girls. Among the other findings that the Study Group reported was that 7 of the 17 
programs it assessed were no longer in operation, primarily because the initial grants 
that supported their operations were not renewed. 

Based on its review of girls’ delinquency programs, the Study Group reported several 
conclusions and recommendations. Among these conclusions and recommendations is 
the need for evaluations and support of girls’ delinquency programs. In particular, the 
Study Group found that insufficient funding has been provided for evaluations of girls’ 
delinquency programs, so definitive conclusions of what works for girls cannot be made. 
Further, the Study Group found that additional, methodologically rigorous evaluations of 
girls’ delinquency programs are needed in order to identify effective and promising 
programs and models that could be replicated at the state and local levels. While the 
Study Group did not specifically quantify the funding needed to support these 
evaluations, it did note that federal sources for evaluation funding and partnerships with 
local colleges and universities are needed. The Study Group also concluded that 
programs need technical assistance to help them prepare for evaluations. Lastly, the 
group found that girls’ delinquency programs that are based on evidence of promising 
techniques should be supported and expanded. In particular, the Study Group 
highlighted program sustainability as an issue, stating that funding needs to be provided 
to ensure that the most promising programs continue to operate after their initial funding 
period is over so that practitioners and policymakers can continue to implement them. 

OJJDP Plans to Provide Technical Assistance to Help Programs, but Could More Fully 
Develop Plans for Supporting Evaluations to Address Its Goals to Prevent and Reduce 
Girls’ Delinquency 

OJJDP has plans to provide technical assistance to girls’ delinquency programs; however, 
its plans for supporting evaluations could be more fully developed to help OJJDP reach 
its goals for addressing girls’ delinquency issues. OJJDP’s goals for addressing girls’ 
delinquency, as stated in the Study Group program announcement, are to identify 
effective and promising programs, program elements, and implementation principles and 
support the development of program models to prevent and reduce girls’ delinquency. 
According to OJJDP officials, in response to the group’s finding about the need to better 
prepare programs for evaluation, the office plans to work with the Study Group and 
using the remainder of its funding — approximately $300,000 — provide a technical 
assistance workshop in October 2009 to help about 10 girls’ delinquency programs 
prepare to be evaluated. In this workshop, OJJDP and the Study Group plan to provide 
information to programs about how evaluations are designed and conducted, how to 
identify appropriate performance measures, and how to collect data needed for program 
evaluators to assess outcomes. OJJDP officials stated that the programs are to be 
selected for participation through an application process and have to meet minimum 
criteria, including having experience working with girls and the capability to collect 
program outcome data. OJJDP officials noted that they intend to limit participation in 
the workshop to about 10 programs to ensure that the programs that are selected receive 
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technical assistance that is targeted to their specific needs. This assistance, according to 
OJJDP officials, will help ensure that when programs do undergo evaluations — whether 
funded by OJJDP, another federal agency, or an independent research organization — the 
evaluations will be more likely to lead to conclusive findings on program effectiveness. 

In addition to providing girls’ delinquency programs with training and technical 
assistance, OJJDP officials also described their plan to fund evaluations of girls’ 
delinquency programs. OJJDP officials stated that as a result of the Study Group’s 
findings along with feedback they received from members of the girls’ delinquency field, 
they recognized the need for evaluations of girls’ delinquency programs. OJJDP officials 
stated that they recognized the need for evaluation in fiscal year 2007 but at the time 
lacked funding to issue a solicitation for such evaluations. Further, 14 of the 18 girls’ 
delinquency experts that we interviewed (including nine Study Group members) 
emphasized the need for OJJDP leadership in supporting evaluations of girls’ 
delinquency programs to identify effective programs. For example, one expert noted that 
since the Study Group found that few programs had been studied, OJJDP would be doing 
a disservice to the girls’ delinquency field if it did not fund rigorous evaluations and help 
programs partner with research organizations. According to the OJJDP officials, the 
office’s goal is to issue a solicitation in early fiscal year 2010 for researchers to apply for 
funding to conduct evaluations of two to five girls’ delinquency programs. These 
evaluations, according to OJJDP officials, are to focus on girls’ delinquency programs 
that have been in operation for a number of years and have data to support evaluations. 
The officials also stated that the planned solicitation would require researchers to 
conduct studies that involve either randomized controlled or quasi-experimental 
research designs. 

OJJDP officials stated that they expect to fund evaluations using the portion of 
appropriation accounts that has been available for research and evaluations, and noted 
that the number of evaluations to be allocated funding depends, in part, on the number of 
applications received, the total available funding, as well as other competing research 
needs and goals. While OJJDP has not yet received an appropriation for fiscal year 2010, 
OJJDP used approximately $12 million in fiscal year 2007 and $14 million in fiscal year 
2008 to support research and evaluations from accounts eligible to support research and 
evaluations of girls’ delinquency programs. OJJDP officials stated that they used this 
funding because in recent years they have not received an appropriation for programs 
and activities authorized under Part D, which is specifically designated for research and 
evaluation, but if they were to receive a Part D appropriation they could increase the 
number of evaluations funded. While OJJDP officials verbally described the planned 
evaluations and funding, they did not provide us with written documentation of the 
planned solicitation because, as of June 2009, it was in draft and subject to change. 

OJJDP officials have described actions they plan to take to respond to the Study Group’s 
findings, and OJJDP reported that these findings will provide a foundation for creating a 
comprehensive program of information dissemination, training, technical assistance, and 
programming to help prevent and reduce girls’ delinquency. However, the office has not 
developed a plan that is documented, is shared with key stakeholders, and includes time 
frames and specific funding requirements and commitments for meeting its girls’ 
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delinquency goals. According to OJJDP officials, they have not developed such a plan 
because the office is in transition and is in the process of initiating efforts to develop an 
officewide research plan, but they are taking steps to address their girls’ delinquency 
goals, for example, through the workshop and planned evaluations. Standard practices 
for program and project management state that specific desired outcomes or results 
should be conceptualized, defined, and documented in the planning process as part of a 
road map, along with the appropriate projects needed to achieve those results, 
supporting resources, and vc^lestones!^^ Standards for Internal Control in the Federal 
Government states that program managers should ensure that there are adequate means 
of obtaining information from and communicating with external stakeholders who may 
have a significant impact on the agency achieving its goals.^° We have also previously 
reported that critical to guiding evaluation and research efforts on a national level is a 
strategy that outlines a process for funding and conducting rigorous evaluations and 
research, identifies the resources needed to achieve it, and assigns accountability for 
accomplishing these actions.^^ In that regard, developing a plan that provides a road map 
to meeting its goals would provide additional assurance that OJJDP’s goals for 
identifying and promoting promising programs and program models would be met and 
communicated to state and local policymakers and practitioners responsible for 
implementing programs to prevent and reduce girls’ delinquency. 

Conclusions 

Preventing and responding to girls’ delinquency have been a concern for federal, state, 
and local governments as well as private and nonprofit juvenile justice organizations for 
over a decade, and the most recent statistics show that girls’ involvement in the juvenile 
justice system is not stabilizing or declining. While OJJDP has undertaken a 6-year, 

$2.6 million study group effort to learn about effective and promising girls’ delinquency 
programs, the lack of rigorous studies of such programs meant that the group was unable 
to identify and promote effective programs and to develop program models to be 
supported at state and local levels. In response to these findings, OJJDP has taken steps 
to provide technical assistance to programs to help prepare them for evaluations and has 
described plans for funding evaluations of girls’ delinquency programs. While these steps 
are consistent with OJJDP’s stated goals, the office lacks a comprehensive documented 
plan that includes time frames and specific funding requirements and commitments for 
meeting its girls’ delinquency goals that it can share with stakeholders. As the Study 
Group plans to conclude its efforts in spring 2010, OJJDP is planning to help ensure the 



“Project Management Institute, The Standard for Program Management. 

'^‘’GAO/AIMD-00-21.3.1. 

^‘GAO, South Florida Ecosystem Restoration: A Strategic Plan and a Process to Resolve Conflicts Are 
Needed to Keep the Effort on Track, GAO/T-RCED-99-170 (Washington, D.C.: Apr. 29, 1999); South Florida 
Ecosystem Restoration: Substantial Progress Made in Developing a Strategic Plan, but Actions Still 
Needed, GAO-01-361 (Washington, D.C.: Mar. 27, 2001); and Great Lakes: A Coordinated Strategic Plan 
and Monitoring System Are Needed to Achieve Restoration Goals, GAO-03-999T (Washington, D.C.: July 
16, 2003). 
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development of effective girls’ delinquency programs and program models by providing 
training and technical assistance to help these programs plan for future evaluations. 
Moreover, such action better positions OJJDP in ensuring that funding for such programs 
is directed to those that are effective in preventing girls’ delinquency and intervening 
after girls have entered the juvenile justice system. As states are continuing to make 
determinations about how to allocate their formula and block grants, and OJJDP 
continues to provide funding to programs through some of its discretionary grant 
programs, information about promising or effective programs and program models could 
help guide these resource decisions. Developing a plan with time frames that clearly 
articulates the office’s approach to its evaluation efforts, including available resources 
needed and committed toward implementing that plan, would help OJJDP ensure that its 
goals to support the development of effective programs are met, and sharing that plan 
with stakeholders would help demonstrate federal leadership to the girls’ delinquency 
field. 

Recommendation for Executive Action 

To help ensure that OJJDP meets its goals to identify effective or promising girls’ 
delinquency programs and supports the development of program models, we recommend 
that the Administrator of OJJDP develop and document a plan that (1) articulates how 
the office intends to respond to the program findings of the Study Group, (2) includes 
time frames and specific funding requirements and commitments, and (3) is shared with 
key stakeholders. 

Agency Comments and Our Evaluation 

We requested comments on a draft of this report from the Attorney General. On July 16, 
2009, we received written comments from OJP, which are reprinted in the enclosure. 

OJP agreed with our recommendation and stated that OJJDP has always intended to 
respond to the findings of the Study Group. OJP described efforts planned in response to 
the findings of the Study Group, including a technical assistance workshop and 
evaluations, which we have discussed in our report. OJP also stated that subsequent 
refined plans and related funding commitments will be based on the outcome of these 
activities and noted that OJJDP, in accordance with the Act, will publish these program 
plans in the Federal Register for review and comment by key stakeholders as well as 
members of the public by December 2009. 

We recognize that OJJDP’s planned activities represent a worthwhile step in responding 
to the findings from the Study Group effort, and are encouraged that OJJDP intends to 
publish a program plan, to include how it will address girls’ delinquency issues. However, 
it is important to note that while OJJDP has been required to publish a program plan 
annually according to the Act, it has not done so since 2002.^^ Following through on its 
current pledge to issue such a plan by December of this year will help provide OJJDP 
with reasonable assurance that it has a well-thought-out approach to ensure that its goals 
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for preventing and reducing girls’ delinquency are met. We also continue to maintain that 
it will be important for this plan to include more than a list of activities in response to the 
Study Group’s findings as OJJDP describes in commenting on this report. Specifically, 
the plan should serve as a road map for OJJDP’s approach for responding to the Study 
Group’s findings, establish overall time frames as well as those for each activity, specify 
funding requirements and associated commitments, and integrate the input of key 
stakeholders, such as girls’ delinquency practitioners. Publishing and implementing such 
a plan would help OJJDP ensure that it meets the goal it articulated at the beginning of 
the 6-year Study Group effort — ^to identify effective and promising programs, program 
elements, and implementation principles and to support the development of program 
models to prevent and reduce girls’ delinquency. 



We are sending copies of this report to interested congressional committees, the 
Attorney General, and other interested parties. In addition, this report will be available at 
no charge on GAO’s Web site at http://www.gao.gov. 

If you or your staff have any questions concerning this report, please contact me at (202) 
512-6510 or larencee@gao.gov . Contact points for our Offices of Congressional Relations 
and Public Affairs may be found on the last page of this report. Mary Catherine Hult, 
Assistant Director; David Alexander; Elizabeth Blair; Amy Brown; Kevin Copping; 
Katherine Davis; Dawn Locke; and Janet Temko made key contributions to this report. 

Sincerely yours. 




Eileen Regen Larence 
Director, Homeland Security 
and Justice Issues 



Enclosure 
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Enclosure 



Comments from the Department of Justice 



U.S. Department of Justice 

Office of Justice Programs 

Office of Audit, Assessment, and Management 



Washington. D.C. 20531 

rjl}ii' 62 Qm 



Ms. Eileen R. Larence 

Director, Homeland Security and Justice Issues 
Government Accountability Office 
441 G Street, NW 
Washington, DC 20548 

Dear Ms. Larence; 

ITiank you for the opportunity to comment on the draft Government Accountability 
Office (GAO) letter report entitled “Juvenile Justice; Technical Assistance and Better 
Defined Evaluation Plans Will Help to Improve Girls’ Delinquency Programs” (GAO-09- 
721R), The Office of Justice Programs agrees with the Recommendation for Executive 
Action, which is restated in bold text below and is followed by our response. 

To help ensure that OJJDP meets its goals to identify effective or promising girls’ 
delinquency programs and supports the development of program models, we 
recommend that the Administrator of OJJDP develop and document a plan, that 
(1) articulates how the agency intends to respond to the program findings of the 
Girls Study Group, (2) includes time frames and specific funding requirements and 
commitments, and (3) is shared with key stakeholders. 

It has always been the intention of the Office of Juvenile Justice and Delinquency 
Prevention (OJJDP) to respond to the findings of the Girls’ Study Group. As discussed 
with the GAO during the review, OJJDP has efforts underway, as well as planned 
initiatives to address the findings. These planned initiatives, and timeframes for 
implementation, are described below. Further planning and funding commitments will be 
based on the outcome of these activities. 

• Hands-On Evaluation Technical Assistance Workshop for Girls’ Delinquency 
Programs 

The workshop is scheduled for October 28-30, 2009, in Chapel Hill, NC. The goal of 
the workshop is to better equip programs to conduct rigorous evaluations of their 
interventions. Unlike general workshops, the Girls’ Study Group Evaluation 
Technical Assistance Workshop will tailor instruction specifically to address the 
needs of participating programs. The faculty at the workshops will be highly skilled 
in evaluation methodology, program development, and strategies on how to partner 
with evaluation professionals. Each participant will leave the workshop with a 
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customized concrete plan for ‘next steps’ and upon request will receive an additional 
hour of technical assistance by phone following the workshop. 

Eligibility will be limited to programs that provide gender-responsive (^linquency 
prevention or interventions for girls, and who have some level of evaluation 
experience. Organizations that have more than one distinct gender responsive 
program are eligible to submit more than one application. There will be a two-phase 
application process. The first phase will require the submission of general program 
information and a description of current evaluation history /experience. Program 
applications will be reviewed based on their program type and evaluation needs; and 
approximately 15 to 20 programs will be invited to proceed to the next application 
phase. Those applicants selected for the second phase will be asked to submit more 
detailed information on the evaluation needs of their programs and reports or findings 
based on previous evaluation work. The workshop organizers will use this 
information to determine which programs provide the best fit between evaluation 
needs and faculty expertise. Approximately 10 programs will ultimately be invited to 
participate in the workshop. Selection of participants for the workshop will be 
completed by the end of September 2009. 

• Enhancement of OJJDP’s Current Girls Delinquency Training and 
Technical Assistance Curriculum 

Using a panel of experts (including staff iiom the Girls’ Study Group, among 
others), OJJDP will update, enhance, and revise the existing Training and 
Technical Assistance Curriculum for Girls’ Delinquency Programming. The 
targeted completion date is December 2009. 

• Release of the FY 2010 Evaluation of Girls’ Delinquency Programs 
Solicitation 

This solicitation will be released, pending availability of funds, for the purpose of 
encouraging partnerships between girls’ delinquency programs and evaluators, 
and providing funding for experimental and quasi-experimental evaluations of 
girls’ delinquency programs. 

In response to the findings of the Girls’ Study Group, OJJDP has plaiuied the above 
described initiatives. Subsequent refined plans and related funding commitments will be 
based on the outcome of these initiatives. As mandated by the Juvenile Justiee and 
Delinquency Prevention Act, the OJJDP will publish these program plans in the Federal 
Register for review and comment by key stakeholders as well as members of the public, 
OJJDP anticipates publishing the program plan in the Federal Register by December 
2009. 



2 
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If you have any questions regarding this response, you or your staff may contact Maureen 
Henneberg, Director, Office of Audit, Assessment, and Management, on (202) 616-3282. 

Sincerely, 



Laurie O. Robinson 

Acting Assistant Attorney General 



Beth McGarry 

Deputy Assistant Attorney for Operations and Management 

Jeffrey Slowikowski 
Acting Director 

Office of Juvenile Justice and Delinquency Prevention 

Maureen Henneberg 
Director 

Office of Audit, Assessment, and Management 

LeToya A. Johnson 

Audit Liaison 

Office of Justice Programs 

Richard P. Theis 
Audit Liaison 
Department of Justice 



(440719) 
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This is a work of the U.S. government and is not subject to copyright protection in the 
United States. The pubiished product may be reproduced and distributed in its entirety 
without further permission from GAO. However, because this work may contain 
copyrighted images or other materiai, permission from the copyright holder may be 
necessary if you wish to reproduce this material separately. 
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